Skip to content

Stamp a node's spend on its error event, so an overspent run can say so - #84

Merged
Shashankss1205 merged 1 commit into
mainfrom
fix/issue-60
Aug 4, 2026
Merged

Stamp a node's spend on its error event, so an overspent run can say so#84
Shashankss1205 merged 1 commit into
mainfrom
fix/issue-60

Conversation

@Shashankss1205

Copy link
Copy Markdown
Collaborator

Fixes #60.

The defect

Tokens were attributed from end events. A node the budget interrupts emits error instead, and error carried no token count — so the audit trail lost the spend in exactly the case an auditor cares about most.

Two runs of the same one-node graph and the same scripted model, differing only in the budget:

before                                          after
budget NOT hit : metrics.tokens=51  errors=0    metrics.tokens=51  errors=0
budget HIT     : metrics.tokens=0   errors=1    metrics.tokens=51  errors=1
                 ^^^^^^^^^^^^^^^^                ^^^^^^^^^^^^^^^^^
enforcement said: "max_tokens reached (51/5)"

The run was killed for spending 51 tokens and reported spending none.

The fix

Every error event is stamped with ctx.meter.tokens - tokens_before — what that node spent — exactly as end is. Four sites: the write/type/routing refusal and the token-ceiling refusal in _leave, and the body-raised handler in each of the sync and async wrappers. The pre-start budget refusal is left alone: no node ran, so there is nothing to attribute.

metrics.summarize adds errors to its measured set, and ReplayedRun.tokens counts failed executions rather than only ok ones.

On double-counting, since that is the thing to get wrong here: NodeExecution.sub_events are documented as a breakdown of the node total, not an addition to it, and orphan_sub_events are by construction the ones no execution claimed. That is what makes ends + orphans safe today, and it applies to error identically. The suite's RunCost.tokens == RunMetrics.tokens assertion still passes, and the new test asserts it directly.

Tests

test_a_run_stopped_for_overspending_reports_what_it_spent — parses the figure out of the enforcement message and asserts the audit trail agrees, plus the cost/metrics equality. Confirmed red with the three source files stashed.

docs/cookbook/01-basics.md documents what each phase carries; its transcript and prose were re-recorded, and tests/test_cookbook_basics.py updated to match. The page catching this is the page doing its job.

Verification

pytest1845 passed, 12 deselected. ruff check grapharc tests → clean.

🤖 Generated with Claude Code

Tokens were attributed from `end` events only. A node the budget interrupts
emits `error` instead, so the spend that triggered enforcement vanished from
the run's own account of itself: `grapharc metrics` said `tokens: 0` for a run
stopped with `max_tokens reached (51/5)`.

Every `error` event now carries what its node spent, exactly as `end` does,
and both `summarize` and the cost report count it. Sub-events inside a node
stay a breakdown of its total rather than an addition, so the disjointness
that kept `ends + orphans` from double-counting holds unchanged, and the
`RunCost.tokens == RunMetrics.tokens` invariant the suite asserts still does.

The basics cookbook stated what each phase carries and had to be re-recorded,
which is the page doing its job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Shashankss1205
Shashankss1205 merged commit 74a4de8 into main Aug 4, 2026
6 checks passed
@Shashankss1205
Shashankss1205 deleted the fix/issue-60 branch August 4, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

observability: a run stopped for overspending reports 0 tokens — the fatal spend is missing from its own audit trail

1 participant